Video Data Research Articles

Breaking the silence: understanding teachers’ use of silence in classrooms

ABSTRACT Silence in classrooms is an undervalued and understudied phenomenon. There is limited research on how teachers behave and think during teachers’ silence in lessons. There are also methodological constraints due to the lack of teacher’s talk during silence. This study used eye-tracking technology to visualize the noticing patterns of two science teachers during silence lasting more than three seconds. Using video data recorded from cameras and eye trackers, we examined each silent event and interpreted teachers’ perceptions and interpretations with consideration of eye fixations, actions of students and teachers during the silence, and teachers’ actions immediately after they broke the silence. We further examined expert-novice differences in teachers’ use of silence. Four categories of teachers’ silence were identified: silence for (1) preparing the classroom for learning; (2) teaching, questioning, and facilitating learning; (3) reflecting and thinking, and (4) behavioural management. Expert-novice differences were identified, especially in the teachers’ use of silence for approaches to teaching, reflection, and behavioural management. The novel contribution of this paper lies in the characterization of silences as observed in actual classroom settings as well as the methodological innovation in using eye trackers and video to overcome the constraints of lack of talk data during silence.

Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features

Current methods for assessing individual well-being in team collaboration at the workplace often rely on manually collected surveys. This limits continuous real-world data collection and proactive measures to improve team member workplace satisfaction. We propose a method to automatically derive social signals related to individual well-being in team collaboration from raw audio and video data collected in teamwork contexts. The goal was to develop computational methods and measurements to facilitate the mirroring of individuals’ well-being to themselves. We focus on how speech behavior is perceived by team members to improve their well-being. Our main contribution is the assembly of an integrated toolchain to perform multi-modal extraction of robust speech features in noisy field settings and to explore which features are predictors of self-reported satisfaction scores. We applied the toolchain to a case study, where we collected videos of 20 teams with 56 participants collaborating over a four-day period in a team project in an educational environment. Our audiovisual speaker diarization extracted individual speech features from a noisy environment. As the dependent variable, team members filled out a daily PERMA (positive emotion, engagement, relationships, meaning, and accomplishment) survey. These well-being scores were predicted using speech features extracted from the videos using machine learning. The results suggest that the proposed toolchain was able to automatically predict individual well-being in teams, leading to better teamwork and happier team members.

All eyes on the signal? - Mapping cohesive discourse structures with eye-tracking data of explanation videos

In this paper, we consider the issue of how the fine-grained multimodal design of educational explanation videos, such as those widely available on YouTube and other platforms, may be made accessible to empirical studies of reception and effectiveness. This is necessary because previous research has often led to conflicting conclusions concerning the roles of particular design elements. We argue that this may largely be due to insufficient characterizations of multimodal design itself. To achieve tighter control of this potential source of variation, we present a multimodal descriptive annotation framework drawing on multimodal (cohesive) film discourse analysis. This framework is seen as a critical first step toward being able to highlight just those differences in design that have functional consequences. For such consequences to accrue, however, viewers need to attend differently to corresponding design differences. The goal of the current paper, therefore, is to use eye-tracking techniques to explore the extent to which discourse structures revealed by our analytic framework relate to recipients' attention allocation. We hypothesize that any potentially emerging anomalies in regards to discourse organization, such as instances of unsuccessful cohesion signaling, may have correlations in the behavioral data. We report our current state of development for performing this kind of multimodal cohesion analysis and some of the unresolved challenges raised when considering how such analyses may be related to performance data.

Performance Evaluation of Congestion Control Algorithm in TCP/IP Differentiated Services During COVID-19 Pandemic Due to Heavy Surge in Real-Time Traffic

Numerous nations implemented countrywide lockdowns because of the COVID-19 epidemic, which forced almost all peoples to remain at home. The imposition of confinement restrictions led to an increase in residential users' Internet traffic demands, particularly for distant work, entertainment, commerce, and education. This resulted in changes to the traffic patterns within the Internet core. In this research, we investigate the impact of the countrywide lockdowns on internet traffic based on dscp values by collecting data from class-A, class-B and Class-C ISP.(Three Class-A ISP, Three Class-B ISP and Three Class-C ISP) which leads to heavy congestion in the network and finally the performance valuation is carried out. The overall rise is exponential in nature, we notice that the traffic volume climbed by 15-30% almost in a quarter, which is a significant gain in such a short amount of time. Direct examination of the traffic sources reveals that, although hypergiants continue to account for a sizeable portion of traffic, there has been a greater increase in non-hypergiant traffic as well as in traffic from home-use applications like Web conferencing, VPNs, and gaming. Academic networks show significant overall declines whereas many networks, particularly those serving residential users, pharmaceutical companies, online interaction platform increases in traffic demands. However, we can see significant gains in these networks when we look at applications related to lecturing and remote working. The changes in the nature of use by customers show that there is a sharp difference in application pertaining to usage. Previous to this period, most usage is on YouTube, Facebook and google search. However, after this period most traffic diverted to online platforms such as zoom, google meet, Teams, and others as well as OTT platform comes into the picture. Over the past two decades, there has been a notable shift in user profiles. In the early 2010s, we saw most traffic was inclined on stored video and audio data in continuation with conventional traffic [23, 49, 66], which was followed by content delivery and streaming applications [7, 24, 35, 37, 52], and mobile applications [32, 67] in the 2010s. Even though user profile modifications are a moving target, they usually take years to complete. So, it was possible to stay current, for example, by measurements. The impact of this pandemic created chaotic conditions as most traffic migrated to real time surge data. Thus, the congestion management becomes more complicated as of previous challenges. The ISP forced to provide more bandwidth to real-time traffic which leads to overprovisioning. Thus, an efficient algorithm is required for managing such surge traffic which may especially meet real time traffic as required by most customers. This paper thus focuses on impact of after corona-19 pandemic effect, the shift of traffic by various users and applications the behavior of some existing algorithms in terms of congestion management and finally suggested a balanced method for managing congestion in such heavy surge traffic.

Sounding out horse mussel sediment thickness: an integrated data approach

Horse mussel beds are listed as a Priority Marine Feature (PMF) in Scotland for their influence in the creation of diverse benthic communities and provision of ecosystem services. In Scotland, horse mussel beds are also recognised for their importance in marine carbon sequestration. Unfortunately, there is a paucity of data on horse mussel bed carbon stocks and sediment thickness. There are also significant financial and logistical barriers with which to address these data gaps. This study investigates the robustness of Sub-Bottom Profiling (SBP) as a cost-effective method to quantify horse mussel sediment thickness across a landscape. Integrating SBP and Drop-Down Video (DDV) data, this study also details an integrated approach to investigate the links between horse mussel habitat condition and sediment thickness. With the addition of abiotic data, this study uses Structural Equation Modelling (SEM) to elucidate key relationships between abiotic factors, biotic variables, and sediment thickness. There is a significant positive correlation of horse mussel habitat condition and sediment thickness. Average horse mussel total sediment thickness, across all measures of habitat condition, was 1.37m. These findings substantially increase previous estimates of horse mussel sediment thickness, and potential value to climate change mitigation through blue carbon frameworks. This study highlights the importance of both abiotic and biotic factors on marine carbon sediment quantification.

Qualitative Research of Robot-Helping Behaviors in a Field Trial

During the previous field study with a robot and its interaction with mall visitors, we observed a surprising event during which a leaflet-distributing robot was abused, although it was subsequently helped by one of its previous abusers. After analyzing 72.25 hours of video data, we identified 47 cases where a robot dropped a leaflet and classified them according to following three criteria: 1) interaction between the potential helper or others with the robot before it dropped the leaflet, 2) the nature of the interaction (abused or not), and 3) whether it was helped. Using the Trajectory Equifinality Model (TEM), we analyzed 19 cases where the robot was helped. We identified the following interaction process that started with individuals who paid attention to the robot, whether they had abusive or non-abusive interactions with it, whether they noticed its failure, and finally whether they helped it. The presence of others encouraged the person to focus on the robot, and the interactions with it led to helping, regardless whether the interaction was abusive. The absence of others when the robot dropped the leaflet encouraged helping. The findings of this study will motivate interaction designs for social robots that can leverage human help.

Assessing Feasibility of Phone Video Data in Diagnosing Parkinsonism (P4-3.020)

Assessing Feasibility of Phone Video Data in Diagnosing Parkinsonism (P4-3.020)

Factors influencing EFL teachers’ provision of oral corrective feedback: the role of teaching experience

ABSTRACT This study highlights critical factors influencing English as a Foreign Language (EFL) teachers’ decisions to provide in-class oral corrective feedback (OCF). It explores the interplay between teachers’ individual differences – namely educational background, teaching experience, and additional training – and their decisions regarding OCF. EFL teachers in three Turkish universities each had three hours of their speaking classes recorded. A thematic analysis of stimulated recall interview data revealed that teachers’ decisions were influenced by learner-related, contextual, and teacher-specific factors. The teachers stated that they provided OCF because they believed that learners expected it. Learner-related factors such as emotions and low proficiency served as deterrents to OCF. Regarding the contextual factors, institutional expectations motivated the provision of OCF, while the nature of activities dissuaded teachers from administering OCF. The teacher factor most commonly cited as a stimulus for providing OCF was the inclination to elicit the correct form from students. The teachers were reluctant to provide OCF because they favoured delayed feedback at the end of the activity or lesson. Regression analysis of the video data revealed that only teaching experience contributed significantly to teachers’ decision-making. The role of teaching experience is discussed, and suggestions are made regarding the provision of OCF.

RTAIAED: A Real-Time Ambulance in an Emergency Detector with a Pyramidal Part-Based Model Composed of MFCCs and YOLOv8.

In emergency situations, every second counts for an ambulance navigating through traffic. Efficient use of traffic light systems can play a crucial role in minimizing response time. This paper introduces a novel automated Real-Time Ambulance in an Emergency Detector (RTAIAED). The proposed system uses special Lookout Stations (LSs) suitably positioned at a certain distance from each involved traffic light (TL), to obtain timely and safe transitions to green lights as the Ambulance in an Emergency (AIAE) approaches. The foundation of the proposed system is built on the simultaneous processing of video and audio data. The video analysis is inspired by the Part-Based Model theory integrating tailored video detectors that leverage a custom YOLOv8 model for enhanced precision. Concurrently the audio analysis component employs a neural network designed to analyze Mel Frequency Cepstral Coefficients (MFCCs) providing an accurate classification of auditory information. This dual-faceted approach facilitates a cohesive and synergistic analysis of sensory inputs. It incorporates a logic-based component to integrate and interpret the detections from each sensory channel, thereby ensuring the precise identification of an AIAE as it approaches a traffic light. Extensive experiments confirm the robustness of the approach and its reliable application in real-world scenarios thanks to its predictions in real time (reaching an fps of 11.8 on a Jetson Nano and a response time up to 0.25 s), showcasing the ability to detect AIAEs even in challenging conditions, such as noisy environments, nighttime, or adverse weather conditions, provided a suitable-quality camera is appropriately positioned. The RTAIAED is particularly effective on one-way roads, addressing the challenge of regulating the sequence of traffic light signals so as to ensure a green signal to the AIAE when arriving in front of the TL, despite the presence of the "double red" periods in which the one-way traffic is cleared of vehicles coming from one direction before allowing those coming from the other side. Also, it is suitable for managing temporary situations, like in the case of roadworks.

'Alright my lovely': The use of terms of endearment as a mitigation device in the care of people living with dementia in the acute hospital environment.

This paper examines how terms of endearment (ToE) are used as a mitigation device in interactions between staff and people living with dementia (PLWD) in the acute hospital environment. ToE are often discouraged in training for healthcare staff. However, this research demonstrates that they are still commonly used in practice. Using conversation analysis, video and audio data were examined to identify the interactional functions of ToE. Analysis showed that ToE play an important role in mitigating potentially face-threatening actions such as when patients are asked to repeat hard-to-interpret talk, or when patient agency is compromised through instruction sequences or having necessary healthcare tasks undertaken. The success of this mitigation is sensitive to the specific interactional circumstances, as well as the responsiveness of the HCP to the patient's voiced concerns. These findings have implications for healthcare practice, training and wider care of PLWD.

Video fragment processing by Ky Fan norm

In this study, we focused on the formalization of video frame descriptions in the context of solving video segmentation problem. Since native video data can have various sizes, dividing each frame into blocks allows present image frame as a square matrix for a formal description. The frame block is a matrix of arbitrary dimensions. The ability to skip the step of matrix transformation to a square dimension or vectorization using some descriptor allows to reduce computational costs, freeing up computational resources required for this transformation. In our study, we used Ky Fan norm value as image frame block descriptor. The Ky Fan norm is built on top of matrix singular values. A singular decomposition does not impose restrictions on either the dimension orthe character of the elements of the original matrix. We conducted a comparative analysis of the effectiveness of the obtained descriptor for different video data sizesand with different aspect ratios, showing that the change in the descriptor for each block is independent of the video sizeand aspect ratios. Changes in the descriptorsfor each block from frame to frame are identical for video data of varying sizes. This means that as a result of such fragment transform, a square matrix of a fixed size iscreated, regardless of the output video size.This makes it possible to unify further processing of the video, which can be useful for the task of information search in large video databases under the conditions of providing a query "ad exemplum". In thiscase, we can analyze the existing database in offline mode and match each video with a fixed square matrix of descriptors, which will significantly reduce the time and amount of resources when matching with the query. Also, this approach can be effectively used to analyze video data for the motion detection and scene change tracking.

Doing More than Confirming: Expanded Responses to Requests for Confirmation in German Talk-in-Interaction

Abstract While requests for confirmation (RfCs) make a yes/no-response relevant, recipients often produce more than a mere confirmation. Our paper explores expanded responses to RfCs in German talk-in-interaction. We focus on responses consisting of a confirmation and an additional TCU/action. Drawing on video data from mundane and institutional settings, we demonstrate how expanded responses are designed and fit the sequential environments in which they occur. We show four different functions fulfilled with expanded responses: (i) specifying and elaborating on the topic introduced in the RfC, (ii) accounting for the intelligibility of the speaker’s prior actions, (iii) resisting the terms of the RfC, and (iv) challenging a RfC by referring to the recipient’s pre-existing knowledge. Finally, we summarize interactional features relevant for the occurrence of expanded responses in our data. In addition, we discuss the implications of our results for future cross-linguistic research.

Edge AI-Based Smart Intersection and Its Application for Traffic Signal Coordination: A Case Study in Pyeongtaek City, South Korea

Recently, smart intersections have emerged as a novel intelligent transportation system (ITS) solution that integrates traffic monitoring, optimal signal control, and even traffic safety. Although smart intersections have been prevalent in many cities, there are a few drawbacks in their practical operations. First, there are inevitable delays in transmitting and processing the video data. Second, there is still a need to develop a real-time signal control method leveraging the acquired data from smart intersections. Thus, this study aims to construct edge AI-based smart intersections and to provide their application for traffic signal coordination. To this end, we install smart intersections on three consecutive intersections of Route 45 in Pyeongtaek city, South Korea. The real-time traffic data are collected by an edge AI video analysis model which is compressed and optimized for its operation in on-site edge devices. The optimized model maintains a similar level of accuracy (93.64%), even if the size is reduced by 97.8% compared to the original. Next, we utilize the LT2 model to treat the coordination failure problem in nonpeak hours occurring unnecessary delays of the side-streets with relatively high demands. We complement some constraint conditions in order to consider the compatibility with the current legacy system. The experiment is conducted on a virtual environment of which geometry and traffic demand are configured based on the features of the study site. The numerical results conclude that the optimal offsets calculated by the LT2 model effectively manage bandwidths for multidirectional flows based on the real-time traffic demands collected from the edge AI-based smart intersections. This study contributes to serve high-resolution real-time traffic data using edge AI on smart intersections and to provide a case study for signal coordination.

Decoding Human Emotions Through Multimodal Analysis Using Deep Learning

In the evolving landscape of emotion recognition, this study addresses the inherent shortcomings in existing technology. Recognizing the imperative role emotions play in human interaction, the need for a more nuanced and accurate multimodal emotion recognition system is evident. This research aims to refine multimodal emotion recognition by addressing existing challenges. The problem statement revolves around enhancing the accuracy and depth of emotion recognition through innovative methodologies. Leveraging Convolutional Neural Networks (CNN), Attention layers, and Bi-LSTM networks, the study utilizes a comprehensive approach. Data fusion involves combining audio extracted from video data, followed by fusion combinations of audio and text. The methodology incorporates SoftMax classifiers for feature recognition. The fusion of audio, text, and video data, coupled with the innovative use of CNN-LSTM and Attention layer networks, contributes to this success. The discussion interprets these results, highlighting the efficacy of the proposed approach in addressing the challenges of multimodal emotion recognition. The outcomes signify a substantial advancement in the field, providing a more comprehensive understanding of emotional expressions.

Application of support vector machine system introducing multiple submodels in data mining

As the information age develops, the scale and form of data become more diverse and diverse. Therefore, people need to use effective means to process information. For large-scale data mining problems, a clustering-based kernel matrix inner product filtering method is introduced to decompose the original quadratic programming problem into multiple sub-problems to support parallel training. And a Spark-based multiple submodels parallel support vector machine is proposed. By introducing open-source tools such as OpenCV, image feature extraction can be performed on large-scale video data. Finally, combined with the designed parallel support vector machine algorithm, video facial and expression recognition is carried out. These experiments confirmed that the research method achieved a maximum acceleration ratio of 2090 times when processing Covtype datasets. The research model could achieve an accuracy of over 99 %. Under the maximum data scale experiment, the research model improved prediction accuracy by 21 percentage points with an acceptable additional time cost of about 4 min only. Task parallel processing could more fully utilize cluster performance, increasing by approximately 3.5 m/s2 from 30 to 150 cores. The research model had the highest recognition accuracy for facial expressions, further demonstrating the effectiveness and superiority of this method. The research method has improved the efficiency of big data analysis and mining, and is of great significance in parallel analysis of video data.

MTVE: Magdeburg tool for video experiments

AbstractMTVE is an open-source software tool (citeware) that can be applied in laboratory and online experiments to implement video communication. The tool enables researchers to gather video data from these experiments in a way that these videos can be later used for automatic analysis through machine learning techniques. The browser-based tool comes with an easy user interface and can be easily integrated into z-Tree, oTree (and other experimental or survey tools). It provides the experimenters control over several communication parameters (e.g., number of participants, resolution), produces high-quality video data, and circumvents the Cocktail Party Problem (i.e., the problem of separating speakers solely based on audio input) by producing separate files. Using some of the recommended Voice-to-Text AI, the experimenters can transcribe individual files. MTVE can merge these individual transcriptions into one conversation.

Field Evaluation of Yellow Change and Red Clearance Intervals with Subsequent Permissive Left-Turn Flashing Yellow Arrow Indication

The flashing yellow arrow (FYA) indication has become commonly used for communicating permissive left-turn operations to road users. A conventional signal phasing communicated to left-turning drivers consists of a leading protected phase followed by a permissive phase using a FYA indication, often referred to as protected-permissive. To transition from the protected to the permissive phase, change (steady yellow arrow) and clearance (steady red arrow, all-red) intervals, or only a change interval, may be used. There is no specific guidance on the use of clearance intervals for left turns with protected-permissive phase and FYA indication. In this study, field data were collected from different geographical regions in the United States to evaluate change and clearance intervals with the objective of developing guidance. Video data were recorded at 37 intersections during peak hours, for approximately 142 h, across eight states. Overall, 4,001 observations of vehicles turning left during the change or clearance interval were analyzed. Field observations were evaluated at the approach level to assess left-turn violation rate and violations per cycle as a function of change and clearance interval configuration. At the individual vehicle level, logistic regression was implemented to evaluate the effect of change interval and clearance interval duration, delayed onset of FYA, and regional variation. The results of the analysis at the approach and individual vehicle level were consistent and indicate that: (i) a clearance interval should be included, (ii) delaying the onset of FYA indication with an extension beyond the all-red clearance interval would reduce left-turn signal violations, and (iii) duration of change intervals has a marginal impact on left-turn signal violations.

A RESEARCH OF THE LATEST APPROACHES TO VISUAL IMAGE RECOGNITION AND CLASSIFICATION

Context. The paper provides an overview of current methods for recognizing and classifying visual images in static images or video stream. The paper will discuss various approaches, including machine learning, current problems of these methods and possible improvements. The biggest challenges of the visual image retrieval and classification task are discussed. The main emphasis is placed on the review of such promising algorithms as SSD, YOLO, R-CNN, an overview of the principles of these methods, network architectures.
 Objective. The aim of the work is to analyze existing studies and find the best algorithm for recognizing and classifying visual images for further activities.
 Method. Primary method is to compare different factors of algorithms in order to select the most perspective one. There are different marks to compare, like image processing speed, accuracy. There are a number of studies and publications that propose methods and algorithms for solving the problem of finding and classifying images in an image [3–6]. It should be noted that most promising approaches are based on machine learning methods. It is worth noting that the proposed methods have drawbacks due to the imperfect implementation of the Faster R-CNN, YOLO, SSD algorithms for working with streaming video. The impact of these drawbacks can be significantly reduced by applying the following solutions: development of combined identification methods, processing of edge cases – tracking the position of identified objects, using the difference between video frames, additional preliminary preparation of input data. Another major area for improvement is the optimization of methods to work with real-time video data, as most current methods focus on images.
 Results. As an outcome of the current research we have found an optimal algorithm for further researches and optimizations.
 Conclusions. Analysis of existent papers and researches has demonstrated the most promising algorithm for further optimizations and experiments. Also current approaches still have some space for further. The next step is to take the chosen algorithm and investigate possibilities to enhance it.

VEpiNet: A multimodal interictal epileptiform discharge detection method based on video and electroencephalogram data

To enhance deep learning-based automated interictal epileptiform discharge (IED) detection, this study proposes a multimodal method, vEpiNet, that leverages video and electroencephalogram (EEG) data. Datasets comprise 24 931 IED (from 484 patients) and 166 094 non-IED 4-second video-EEG segments. The video data is processed by the proposed patient detection method, with frame difference and Simple Keypoints (SKPS) capturing patients’ movements. EEG data is processed with EfficientNetV2. The video and EEG features are fused via a multilayer perceptron. We developed a comparative model, termed nEpiNet, to test the effectiveness of the video feature in vEpiNet. The 10-fold cross-validation was used for testing. The 10-fold cross-validation showed high areas under the receiver operating characteristic curve (AUROC) in both models, with a slightly superior AUROC (0.9902) in vEpiNet compared to nEpiNet (0.9878). Moreover, to test the model performance in real-world scenarios, we set a prospective test dataset, containing 215 h of raw video-EEG data from 50 patients. The result shows that the vEpiNet achieves an area under the precision–recall curve (AUPRC) of 0.8623, surpassing nEpiNet’s 0.8316. Incorporating video data raises precision from 70% (95% CI, 69.8%–70.2%) to 76.6% (95% CI, 74.9%–78.2%) at 80% sensitivity and reduces false positives by nearly a third, with vEpiNet processing one-hour video-EEG data in 5.7 min on average. Our findings indicate that video data can significantly improve the performance and precision of IED detection, especially in prospective real clinic testing. It suggests that vEpiNet is a clinically viable and effective tool for IED analysis in real-world applications.

Building Depth Maps Using an Active-Pulse Television Measuring System in Real Time Domain

The paper presents the results of software development for building depth maps based on video data from a television camera of an active-pulse television measuring system (AP TMS) in real time domain. The development of such software is required to conduct various scientific studies, as well as to improve the methods and techniques for building depth maps and remote measurement of the characteristics of objects of interest. The software was implemented using the Python programming language with additional libraries installed. According to the results of testing the implemented algorithm, it was found that the calculation speed using the graphics processing unit (GPU) is on average 3.5 times higher than the speed of the algorithm using only the central processing unit (CPU). It has been established that with the help of CUDA cores it is possible to build depth maps in real time domain at the maximum possible resolution of video frames of the system (1544x2064 pixels), while when using the central processor, real-time operation is possible only at a reduced resolution of video frames (772x1032 pixels).